{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "lesbian-springer",
   "metadata": {},
   "source": [
    "## Huggingface Speech to Sentiment Pipeline Example\n",
    "\n",
    "In this example we create a Pipeline to chain two huggingface models to allow speech to sentiment functionalityand add an explainer to understand the result.\n",
    "\n",
    "This example also illustrates how explainers can target pipelines to allow complex explanations flows.\n",
    "\n",
    "![architecture](speech-to-sentiment.jpg)\n",
    "\n",
    "This example requires **ffmpeg** package to be installed locally. run `make install-requirements` for the Python dependencies.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3154774d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywebrtc import AudioRecorder, CameraStream\n",
    "import torchaudio\n",
    "from IPython.display import Audio\n",
    "import base64\n",
    "import json\n",
    "import requests\n",
    "import os\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faeef0f7",
   "metadata": {},
   "source": [
    "Create a method to load speech from recorder; transform into mp3 and send at base64 data. On return of the result extract and show the text and sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "526b0678",
   "metadata": {},
   "outputs": [],
   "source": [
    "reqJson = json.loads('{\"inputs\":[{\"name\":\"args\", \"parameters\": {\"content_type\": \"base64\"}, \"data\":[],\"datatype\":\"BYTES\",\"shape\":[1]}]}')\n",
    "url = \"http://0.0.0.0:9000/v2/models/model/infer\"\n",
    "def infer(resource: str):\n",
    "    with open('recording.webm', 'wb') as f:\n",
    "        f.write(recorder.audio.value)\n",
    "    !ffmpeg -i recording.webm -vn -ab 128k -ar 44100 file.mp3 -y -hide_banner -loglevel panic\n",
    "    with open(\"file.mp3\", mode='rb') as file:\n",
    "        fileContent = file.read()\n",
    "        encoded = base64.b64encode(fileContent)\n",
    "        base64_message = encoded.decode('utf-8')\n",
    "    reqJson[\"inputs\"][0][\"data\"] = [str(base64_message)]\n",
    "    headers = {\"Content-Type\": \"application/json\", \"seldon-model\": resource}\n",
    "    response_raw = requests.post(url, json=reqJson, headers=headers)\n",
    "    j = response_raw.json()\n",
    "    sentiment = j[\"outputs\"][0][\"data\"][0]\n",
    "    text = j[\"outputs\"][1][\"data\"][0]\n",
    "    reqId = response_raw.headers[\"x-request-id\"]\n",
    "    print(reqId)\n",
    "    os.environ[\"REQUEST_ID\"]=reqId\n",
    "    print(base64.b64decode(text))\n",
    "    print(base64.b64decode(sentiment))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "026afc59",
   "metadata": {},
   "source": [
    "### Load Huggingface Models\n",
    "\n",
    "We will load two Huggingface models for speech to text and text to sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "43b7a0a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: whisper\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/mlserver/huggingface/whisper\"\n",
      "  requirements:\n",
      "  - huggingface\n",
      "---\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: sentiment\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/mlserver/huggingface/sentiment\"\n",
      "  requirements:\n",
      "  - huggingface\n"
     ]
    }
   ],
   "source": [
    "!cat ../../models/hf-whisper.yaml\n",
    "!echo \"---\"\n",
    "!cat ../../models/hf-sentiment.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9aacd53d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ../../models/hf-whisper.yaml\n",
    "!seldon model load -f ../../models/hf-sentiment.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b13a6082",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status whisper -w ModelAvailable | jq -M .\n",
    "!seldon model status sentiment -w ModelAvailable | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4149096",
   "metadata": {},
   "source": [
    "### Create Explain Pipeline\n",
    "\n",
    "To allow Alibi-Explain to more easily explain the sentiment we will need:\n",
    "\n",
    " * input and output transfrorms that take the Dict values input and output by the Huggingface sentiment model and turn them into values that Alibi-Explain can easily understand with the core values we want to explain and the outputs from the sentiment model.\n",
    " * A separate Pipeline to allow us to join the sentiment model with the output transform\n",
    "\n",
    "These transform models are MLServer custom runtimes as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6d64d3bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[37m# Copyright (c) 2024 Seldon Technologies Ltd.  \u001b[39;49;00m\n",
      "\n",
      "\u001b[37m# Use of this software is governed by \u001b[39;49;00m\n",
      "\u001b[37m# (1) the license included in the LICENSE file or \u001b[39;49;00m\n",
      "\u001b[37m# (2) if the license included in the LICENSE file is the Business Source License 1.1, \u001b[39;49;00m\n",
      "\u001b[37m# the Change License after the Change Date as each is defined in accordance with the LICENSE file.\u001b[39;49;00m\n",
      "\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m MLModel\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mtypes\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m InferenceRequest, InferenceResponse, ResponseOutput\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mcodecs\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mstring\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m StringRequestCodec\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mlogging\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m logger\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mjson\u001b[39;49;00m\n",
      "\n",
      "\n",
      "\u001b[34mclass\u001b[39;49;00m \u001b[04m\u001b[32mSentimentInputTransformRuntime\u001b[39;49;00m(MLModel):\n",
      "\n",
      "  \u001b[34masync\u001b[39;49;00m \u001b[34mdef\u001b[39;49;00m \u001b[32mload\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m) -> \u001b[36mbool\u001b[39;49;00m:\n",
      "    \u001b[34mreturn\u001b[39;49;00m \u001b[36mself\u001b[39;49;00m.ready\n",
      "\n",
      "  \u001b[34masync\u001b[39;49;00m \u001b[34mdef\u001b[39;49;00m \u001b[32mpredict\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m, payload: InferenceRequest) -> InferenceResponse:\n",
      "    logger.info(\u001b[33m\"\u001b[39;49;00m\u001b[33mpayload (input-transform): \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,payload)\n",
      "    res_list = \u001b[36mself\u001b[39;49;00m.decode_request(payload, default_codec=StringRequestCodec)\n",
      "    logger.info(\u001b[33m\"\u001b[39;49;00m\u001b[33mres list (input-transform): \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,res_list)\n",
      "    texts = []\n",
      "    \u001b[34mfor\u001b[39;49;00m res \u001b[35min\u001b[39;49;00m res_list:\n",
      "      logger.info(\u001b[33m\"\u001b[39;49;00m\u001b[33mdecoded data (input-transform): \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, res)\n",
      "      \u001b[37m#text = json.loads(res)\u001b[39;49;00m\n",
      "      text = res\n",
      "      texts.append(text[\u001b[33m\"\u001b[39;49;00m\u001b[33mtext\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "\n",
      "    logger.info(\u001b[33m\"\u001b[39;49;00m\u001b[33mtransformed data (input-transform): \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, texts)\n",
      "    response =  StringRequestCodec.encode_response(\n",
      "      model_name=\u001b[33m\"\u001b[39;49;00m\u001b[33msentiment\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "      payload=texts\n",
      "    )\n",
      "    logger.info(\u001b[33m\"\u001b[39;49;00m\u001b[33mresponse (input-transform): \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, response)\n",
      "    \u001b[34mreturn\u001b[39;49;00m response\n"
     ]
    }
   ],
   "source": [
    "!cat ./sentiment-input-transform/model.py | pygmentize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "251ae9ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[37m# Copyright (c) 2024 Seldon Technologies Ltd.  \u001b[39;49;00m\n",
      "\n",
      "\u001b[37m# Use of this software is governed by \u001b[39;49;00m\n",
      "\u001b[37m# (1) the license included in the LICENSE file or \u001b[39;49;00m\n",
      "\u001b[37m# (2) if the license included in the LICENSE file is the Business Source License 1.1, \u001b[39;49;00m\n",
      "\u001b[37m# the Change License after the Change Date as each is defined in accordance with the LICENSE file.\u001b[39;49;00m\n",
      "\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m MLModel\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mtypes\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m InferenceRequest, InferenceResponse, ResponseOutput\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mcodecs\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m StringCodec, Base64Codec, NumpyRequestCodec\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mcodecs\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mstring\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m StringRequestCodec\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mcodecs\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mnumpy\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m NumpyRequestCodec\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mbase64\u001b[39;49;00m\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mmlserver\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mlogging\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m logger\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mnumpy\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mnp\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mjson\u001b[39;49;00m\n",
      "\n",
      "\u001b[34mclass\u001b[39;49;00m \u001b[04m\u001b[32mSentimentOutputTransformRuntime\u001b[39;49;00m(MLModel):\n",
      "\n",
      "  \u001b[34masync\u001b[39;49;00m \u001b[34mdef\u001b[39;49;00m \u001b[32mload\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m) -> \u001b[36mbool\u001b[39;49;00m:\n",
      "    \u001b[34mreturn\u001b[39;49;00m \u001b[36mself\u001b[39;49;00m.ready\n",
      "\n",
      "  \u001b[34masync\u001b[39;49;00m \u001b[34mdef\u001b[39;49;00m \u001b[32mpredict\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m, payload: InferenceRequest) -> InferenceResponse:\n",
      "    logger.\u001b[34minfo\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33mpayload (output-transform): \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,payload)\n",
      "    res_list = \u001b[36mself\u001b[39;49;00m.decode_request(payload, default_codec=StringRequestCodec)\n",
      "    logger.\u001b[34minfo\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33mres list (output-transform): \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,res_list)\n",
      "    scores = []\n",
      "    \u001b[34mfor\u001b[39;49;00m res \u001b[35min\u001b[39;49;00m res_list:\n",
      "      logger.debug(\u001b[33m\"\u001b[39;49;00m\u001b[33mdecoded data (output transform): \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,res)\n",
      "      \u001b[37m#sentiment = json.loads(res)\u001b[39;49;00m\n",
      "      sentiment = res\n",
      "      \u001b[34mif\u001b[39;49;00m sentiment[\u001b[33m\"\u001b[39;49;00m\u001b[33mlabel\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m] == \u001b[33m\"\u001b[39;49;00m\u001b[33mPOSITIVE\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\n",
      "        scores.\u001b[34mappend\u001b[39;49;00m(\u001b[34m1\u001b[39;49;00m)\n",
      "      \u001b[34melse\u001b[39;49;00m:\n",
      "        scores.\u001b[34mappend\u001b[39;49;00m(\u001b[34m0\u001b[39;49;00m)\n",
      "    response =  NumpyRequestCodec.encode_response(\n",
      "      model_name=\u001b[33m\"\u001b[39;49;00m\u001b[33msentiments\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "      payload=np.\u001b[34marray\u001b[39;49;00m(scores)\n",
      "    )\n",
      "    logger.\u001b[34minfo\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33mresponse (output-transform): \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, response)\n",
      "    \u001b[34mreturn\u001b[39;49;00m response\n"
     ]
    }
   ],
   "source": [
    "!cat ./sentiment-output-transform/model.py | pygmentize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "26c94fb2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: sentiment-input-transform\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/scv2/examples/huggingface/mlserver_1.3.5/sentiment-input-transform\"\n",
      "  requirements:\n",
      "  - mlserver\n",
      "  - python\n",
      "---\n",
      "apiVersion: mlops.seldon.io/v1alpha1\n",
      "kind: Model\n",
      "metadata:\n",
      "  name: sentiment-output-transform\n",
      "spec:\n",
      "  storageUri: \"gs://seldon-models/scv2/examples/huggingface/mlserver_1.3.5/sentiment-output-transform\"\n",
      "  requirements:\n",
      "  - mlserver\n",
      "  - python\n"
     ]
    }
   ],
   "source": [
    "!cat ../../models/hf-sentiment-input-transform.yaml\n",
    "!echo \"---\"\n",
    "!cat ../../models/hf-sentiment-output-transform.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "64fc938f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ../../models/hf-sentiment-input-transform.yaml\n",
    "!seldon model load -f ../../models/hf-sentiment-output-transform.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d6cd0078",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "!seldon model status sentiment-input-transform -w ModelAvailable | jq -M .\n",
    "!seldon model status sentiment-output-transform -w ModelAvailable | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2ff58083",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: sentiment-explain\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "    - name: sentiment\r\n",
      "      tensorMap:\r\n",
      "        sentiment-explain.inputs.predict: array_inputs\r\n",
      "    - name: sentiment-output-transform\r\n",
      "      inputs:\r\n",
      "      - sentiment\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - sentiment-output-transform\r\n"
     ]
    }
   ],
   "source": [
    "!cat ../../pipelines/sentiment-explain.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "830d6bb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ../../pipelines/sentiment-explain.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "95fa2faf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"pipelineName\": \"sentiment-explain\",\r\n",
      "  \"versions\": [\r\n",
      "    {\r\n",
      "      \"pipeline\": {\r\n",
      "        \"name\": \"sentiment-explain\",\r\n",
      "        \"uid\": \"cihuo3svgtec73bj6ncg\",\r\n",
      "        \"version\": 2,\r\n",
      "        \"steps\": [\r\n",
      "          {\r\n",
      "            \"name\": \"sentiment\",\r\n",
      "            \"tensorMap\": {\r\n",
      "              \"sentiment-explain.inputs.predict\": \"array_inputs\"\r\n",
      "            }\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"sentiment-output-transform\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"sentiment.outputs\"\r\n",
      "            ]\r\n",
      "          }\r\n",
      "        ],\r\n",
      "        \"output\": {\r\n",
      "          \"steps\": [\r\n",
      "            \"sentiment-output-transform.outputs\"\r\n",
      "          ]\r\n",
      "        },\r\n",
      "        \"kubernetesMeta\": {}\r\n",
      "      },\r\n",
      "      \"state\": {\r\n",
      "        \"pipelineVersion\": 2,\r\n",
      "        \"status\": \"PipelineReady\",\r\n",
      "        \"reason\": \"created pipeline\",\r\n",
      "        \"lastChangeTimestamp\": \"2023-07-04T09:53:19.250753906Z\",\r\n",
      "        \"modelsReady\": true\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status sentiment-explain -w PipelineReady | jq -M ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "229c61af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Model\r\n",
      "metadata:\r\n",
      "  name: sentiment-explainer\r\n",
      "spec:\r\n",
      "  storageUri: \"gs://seldon-models/scv2/examples/huggingface/speech-sentiment/explainer\"\r\n",
      "  explainer:\r\n",
      "    type: anchor_text\r\n",
      "    pipelineRef: sentiment-explain\r\n"
     ]
    }
   ],
   "source": [
    "!cat ../../models/hf-sentiment-explainer.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7ba179b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model load -f ../../models/hf-sentiment-explainer.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a1efedc0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: Model wait status timeout\r\n"
     ]
    }
   ],
   "source": [
    "!seldon model status sentiment-explainer -w ModelAvailable | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dddb4c53",
   "metadata": {},
   "source": [
    "### Speech to Sentiment Pipeline with Explanation\n",
    "\n",
    "We can now create the final pipeline that will take speech and generate sentiment alongwith an explanation of why that sentiment was predicted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d787e1ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: mlops.seldon.io/v1alpha1\r\n",
      "kind: Pipeline\r\n",
      "metadata:\r\n",
      "  name: speech-to-sentiment\r\n",
      "spec:\r\n",
      "  steps:\r\n",
      "    - name: whisper\r\n",
      "    - name: sentiment\r\n",
      "      inputs:\r\n",
      "      - whisper\r\n",
      "      tensorMap:\r\n",
      "        whisper.outputs.output: args\r\n",
      "    - name: sentiment-input-transform\r\n",
      "      inputs:\r\n",
      "      - whisper\r\n",
      "    - name: sentiment-explainer\r\n",
      "      inputs:\r\n",
      "      - sentiment-input-transform\r\n",
      "  output:\r\n",
      "    steps:\r\n",
      "    - sentiment\r\n",
      "    - whisper\r\n"
     ]
    }
   ],
   "source": [
    "!cat ../../pipelines/speech-to-sentiment.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "22d5997a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline load -f ../../pipelines/speech-to-sentiment.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8bad7972",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"pipelineName\": \"speech-to-sentiment\",\r\n",
      "  \"versions\": [\r\n",
      "    {\r\n",
      "      \"pipeline\": {\r\n",
      "        \"name\": \"speech-to-sentiment\",\r\n",
      "        \"uid\": \"cihuqb4vgtec73bj6nd0\",\r\n",
      "        \"version\": 2,\r\n",
      "        \"steps\": [\r\n",
      "          {\r\n",
      "            \"name\": \"sentiment\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"whisper.outputs\"\r\n",
      "            ],\r\n",
      "            \"tensorMap\": {\r\n",
      "              \"whisper.outputs.output\": \"args\"\r\n",
      "            }\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"sentiment-explainer\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"sentiment-input-transform.outputs\"\r\n",
      "            ]\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"sentiment-input-transform\",\r\n",
      "            \"inputs\": [\r\n",
      "              \"whisper.outputs\"\r\n",
      "            ]\r\n",
      "          },\r\n",
      "          {\r\n",
      "            \"name\": \"whisper\"\r\n",
      "          }\r\n",
      "        ],\r\n",
      "        \"output\": {\r\n",
      "          \"steps\": [\r\n",
      "            \"sentiment.outputs\",\r\n",
      "            \"whisper.outputs\"\r\n",
      "          ]\r\n",
      "        },\r\n",
      "        \"kubernetesMeta\": {}\r\n",
      "      },\r\n",
      "      \"state\": {\r\n",
      "        \"pipelineVersion\": 2,\r\n",
      "        \"status\": \"PipelineReady\",\r\n",
      "        \"reason\": \"created pipeline\",\r\n",
      "        \"lastChangeTimestamp\": \"2023-07-04T09:58:04.277171896Z\",\r\n",
      "        \"modelsReady\": true\r\n",
      "      }\r\n",
      "    }\r\n",
      "  ]\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!seldon pipeline status speech-to-sentiment -w PipelineReady | jq -M ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c0800c8",
   "metadata": {},
   "source": [
    "### Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "85884fb2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f68d6e714e9642bc9484a625247379f0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "AudioRecorder(audio=Audio(value=b'', format='webm'), stream=CameraStream(constraints={'audio': True, 'video': …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "camera = CameraStream(constraints={'audio': True,'video':False})\n",
    "recorder = AudioRecorder(stream=camera)\n",
    "recorder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "6b8984ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cihuqm8fh5ss73der5gg\n",
      "b'{\"text\": \" Cambridge is a great place.\"}'\n",
      "b'{\"label\": \"POSITIVE\", \"score\": 0.9998548030853271}'\n"
     ]
    }
   ],
   "source": [
    "infer(\"speech-to-sentiment.pipeline\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d36a92",
   "metadata": {},
   "source": [
    "We will wait for the explanation which is run asynchronously to the functional output from the Pipeline above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "81107af6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "......\n",
      "Explanation anchors: ['great']\n"
     ]
    }
   ],
   "source": [
    "while True:\n",
    "    base64Res = !seldon pipeline inspect speech-to-sentiment.sentiment-explainer.outputs --format json \\\n",
    "          --request-id ${REQUEST_ID}\n",
    "    j = json.loads(base64Res[0])\n",
    "    if j[\"topics\"][0][\"msgs\"] is not None:\n",
    "        expBase64 = j[\"topics\"][0][\"msgs\"][0][\"value\"][\"outputs\"][0][\"contents\"][\"bytesContents\"][0]\n",
    "        expRaw = base64.b64decode(expBase64)\n",
    "        exp = json.loads(expRaw)\n",
    "        print(\"\")\n",
    "        print(\"Explanation anchors:\",exp[\"data\"][\"anchor\"])\n",
    "        break\n",
    "    else:\n",
    "        print(\".\",end='')\n",
    "        time.sleep(1)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "264570be",
   "metadata": {},
   "source": [
    "### Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "6e401645",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon pipeline unload speech-to-sentiment\n",
    "!seldon pipeline unload sentiment-explain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "0dd63ff2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!seldon model unload whisper\n",
    "!seldon model unload sentiment\n",
    "!seldon model unload sentiment-explainer\n",
    "!seldon model unload sentiment-output-transform\n",
    "!seldon model unload sentiment-input-transform"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
